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SUMMARY 


PURPOSE 

The  purpose  of  the  project  described  in  this  report  was  to  conduct 
an  evaluation  comparing  fully  proceduralized  troubleshooting  aids 
(FPTAs) , logic  tree  troubleshooting  aids  (LTTAs)t  and  technical  orders 
(TOs)  in  terms  of: 

1.  The  costs  Involved  in  developing  each  type  of  aid. 

2.  The  technical  accuracy  of  each  type  of  data. 

3.  The  effectiveness  of  each  type  of  data  for  supporting 
maintenance  personnel  of  varying  experience  levels  in 
performing  troubleshooting  tasks. 


METHOD 

Cost  comparison:  The  available  data  on  costs  to  produce  each  of 

the  three  types  of  technical  data  were  obtained  from  FPTA  and  LTTA 
contractors,  and  from  the  Air  Force  Logistic  Centers  responsible  for  TO 
procurement.  Due  to  differences  in  coverage,  developer  experience,  data 
base  and  types  of  systems  for  which  the  technical  data  were  developed, 
difficulty  was  experienced  in  arriving  at  a single  measure  which  would 
permit  a meaningful  comparison  of  the  three  document  types.  The  approach 
selected  was  to  base  the  analysis  on  the  cost  to  produce  a page  unit.  A 
page  unit  was  defined  as  a standard  8 1/2-  by  11-inch  format.  Use  of 
this  standard  made  it  necessary  to  convert  the  A-  by  8-inch  page  format 
used  in  the  LTTA  checkout  procedures  for  organizational  level  maintenance 
into  the  standard  page  unit.  No  other  page  conversions  were  required. 

Technical  accuracy:  Review  of  worldwide  Air  Force  maintenance 

actions  was  undertaken  to  provide  identification  of  maintenance  areas 
which  could  serve  as  representative  candidates  for  development  of 
troubleshooting  test  problems.  Thirty  specific  troubleshooting  test 
problems  were  selected  and  distributed  throughout  the  AN/APN-147  (radar) 
and  AN/ASN-35  (computer)  systems  of  the  C-141A  aircraft.  Each  problem 
was^ Inserted  into  working  system  equipment.  The  effects  on  system 
operation  were  identified  and  a systematic  evaluation  of  the  accuracy  of 
the  technical  documentation  types  in  supporting  the  isolation  of  the 
faults  was  undertaken.  Records  were  maintained  on  the  errors  found  in 
the  data  and  on  whether  the  procedure  led  to  isolation  of  the  problem. 
Those  faults  which  could  be  successfully  isolated  by  each  of  the  three 
types  of  tschnlcal  data  were  retained  for  consideration  as  a test 
problem  in  the  field  data  collection  phase. 

Experimental  evaluation  of  effectiveness  of  each  technical  data 
type:  Three  experience  levels  of  Air  Force  technicians  in  the  Avionics 

Inertial  and  Radar  Navigational  Systems  Specialty  (AFSC  328X4)  served 
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as  subjects.  The  experience  levels  were:  recent  graduates  of  the 

technical  training  course  (no  field  experience);  technicians  with  six 
months  or  less  experience;  and  technicians  with  more  than  six  months 
experience.  Eighteen  technicians  at  each  experience  level  were  used  as 
subjects.  A determination  was  made  that  it  would  not  be  appropriate  to 
test  the  no-experience  group  on  technical  orders.  This  decision  made  it 
necessary  to  use  two  different,  but  similar,  experimental  designs.  Both 
designs  involved  repeated  measures  on  subjects.  In  the  first  design, 
the  no-experlence  group  was  tested  using  two  types  of  data,  FPTAs  and 
LTTAs.  In  the  second  design,  the  two  experienced  groups  of  subjects 
were  tested  using  all  three  types  of  data.  Data  collection  was  accom- 
plished at  Keesler  Technical  Training  Center  and  at  three  Military 
Airlift  Command  bases:  McChord,  Travis,  and  McGuire. 


RESULTS 

The  cost  comparison,  primarily  due  to  the  fact  that  sufficient 
comparable  data  could  not  be  assembled,  produced  inconclusive  results. 
However,  the  available  data  did  suggest  that  the  costs. of  developing 
fully  procedurallzed  troubleshooting  aids  in  a newly  developed  system 
where  all  support  data  would  also  have  to  be  generated  may  not  be  much 
different  from  the  costs  incurred  in  technical  order  development. 

Errors  in  accuracy  were  found  in  both  the  FPTA  and  LTTA.  The 
errors  discovered  vary  in  their  Impact  on  performance  from  negligible  to 
completely  disruptive.  However,  no  accuracy  errors  were  found  that  were 
considered  beyond  solution.  Increased  emphasis  on  quality  control  would 
eliminate  most  of  the  accuracy  errors  found. 

The  results  of  the  experimental  evaluation  clearly  demonstrate  that 
the  use  of  procedurallzed  troubleshooting  aids  resulted  in  significantly 
better  troubleshooting  than  the  use  of  the  TO.  This  finding  held  for 
two  of  the  three  measures:  proportion  of  problems  solved  and  spare 

parts  consumed.  For  the  third  measure,  time  to  troubleshoot,  the  use  of 
TOs  resulted  in  better  performance  at  the  organizational  level  but  not 
at  the  Intermediate  level  of  maintenance. 

In  comparing  the  FPTA  against  the  LTTA,  the  FPTA  was  superior  for 
the  no-experience  group  on  all  measures.  For  experienced  technicians, 
use  of  the  FPTA  resulted  in  slightly  better  performance  on  all  measures 
except  time  to  troubleshoot  at  the  organizational  level. 

In  addition  to  the  objective  performance  data,  subjective  opinion 
data  of  the  technicians  provided  unqualified  support  for  the  use  of  the 
FPTA  as  the  most  favored  of  the  three  types  of  technical  documentation. 
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The  findings  of  this  research  effort  support  the  following  recom- 
mendations: 

1.  Consideration  should  be  given  to  development  of  FPTAs  or 
LTTAs  for  new  procurements  of  technical  data  for  trouble- 
shooting (whether  for  new  or  existing  systems).  It  is 
especially  important  that  these  types  of  data  be  considered 
for  intermediate  level  maintenance  where  the  benefits  are 
likely  to  be  greatest. 


2.  Proceduralized  troubleshooting  data  (FPTA,  LTTA,  or  hybrid) 
should  be  developed  for  a modern  Air  Force  system  for  opera- 
tional use  to  provide  an  opportunity  to  study  the  problems 
associated  with  using  this  type  of  data  in  an  operational 
setting. 

3.  Strong  emphasis  should  be  placed  on  accuracy  evaluation. 

4.  Additional  analyses  of  the  costs  and  cost  benefits  of  FPTAs 
and  LTTAs  should  be  made. 

5.  The  potential  of  FPTAs  and  LTTAs  for  reducing  training  costs 
should  be  evaluated. 

6.  Strong  consideration  should  be  given  to  modification  of  the 
LTTA  format  to  include  more  of  the  proceduralization  material 
contained  in  the  FPTA. 

This  report  is  in  three  volumes:  Volume  I presents  a summary  of 

the  project  and  the  results.  The  present  volume  presents  a detailed 
description  of  the  experimental  procedures  used  in  the  study.  Volume  III 
presents  the  test  administrator's  guide  and  the  job  performance  tests. 


PREFACE 


This  study  was  initiated  by  the  Advanced  Systems  Division,  Air 
Force  Human  Resources  Laboratory  (AFHRL) , Wright-Patterson  Air  Force 
Base,  Ohio.  The  research  was  performed  by  Systems  Research  Laboratories, 
Inc. , Dayton,  Ohio,  under  Contract  F33615-75-C-5103,  with  Dr.  Norman  R. 
Potter  as  principal  investigator. 

The  authors  wish  to  acknowledge  the  contributions  of  the  members  of 
the  Systems  Research  Laboratories  field  evaluation  team,  Mr.  Ned  H. 
Kearns,  Mr.  P.  Richard  Hubbert,  Mr.  James  V.  Landolfi,  and  Mr.  Daniel  L. 
Rice.  Without  their  efforts  and  dedication  to  the  data  collection 
effort,  this  project  could  not  have  been  accomplished.  Further,  an 
appreciation  must  be  expressed  to  MSgt  Bobby  Spaulding,  AFHRL,  for  his 
guidance  and  support  throughout  the  field  data  collection  effort,  and  to 
Mr.  John  J.  Klesch,  AFHRL,  for  his  guidance  and  knowledgeable  consulta- 
tion throughout  the  project. 
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EVALUATION  OF  THREE  TYPES  OF  TECHNICAL  DATA  FOR 
TROUBLESHOOTING:  METHODOLOGY  FOR  FIELD  EVALUATION 


INTRODUCTION 


The  purpose  of  this  effort  was  to  collect  data  on  which  to  base 
objective  decisions  about  the  relative  merits  of  two  new  types  of  technical 
documentation,  fully  proceduralized  troubleshooting  aid  (FPTA)  and  logic 
tree  troubleshooting  aid  (LTTA) , in  comparison  to  the  standard  Air  Force 
technical  order  (TO).  The  major  emphasis  of  this  effort  entailed  the 
collection  of  data  through  the  design  and  conduct  of  a carefully  con- 
trolled systematic  experimental  evaluation  of  troubleshooting  performance 
incorporating  the  use  of  the  three  types  of  technical  documentation  to 
guide  technical  activities. 

Distinct  advantages  have  been  claimed  for  the  use  of  technical 
documentation  other  than  Air  Force  TOs;  however,  little  empirical  data 
can  be  found  to  support,  or  refute,  these  claims.  This  study  was  designed 
to  supply  objective  data  on  which  the  merits  of  FPTAs  and  LTTAs,  in 
comparison  with  TOs,  could  be  weighed. 

The  specific  objectives  of  the  experimental  evaluation  were  to 
obtain  data  which  could  permit  answers  to  the  following  questions: 

1.  Does  FPTA  performance  differ  from  LTTA  performance?  Are  both 
FPTA  and  LTTA  performance  different  from  TO  performance? 

2.  Does  quality  of  performance  with  each  of  the  three  types  of 
documentation  depend  upon  experience  level  of  the  technician? 

Is  there  interaction  between  experience  level  and  type  of 
documentation? 

3.  Does  performance  with  each  of  the  three  types  of  technical 
documentation  differ  for  organizational  and  intermediate 
level  maintenance  troubleshooting?  Is  there  an  interaction 
between  type  of  technical  documentation  and  level  of  mainte- 
nance troubleshooting? 

4.  Is  there  an  Interaction  between  level  of  maintenance  trouble- 
shooting and  level  of  experience?  If  so,  what  is  the  nature 
of  the  Interaction? 

5.  Is  there  an  interaction  among  types  of  technical  documentation, 
experience  level  and  level  of  maintenance  troubleshooting?  If 
ao,  what  is  the  nature  of  the  interaction? 
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As  a preliminary  step  in  the  evaluation  of  the  three  types  of 
technical  documentation  for  troubleshooting  Air  Force  equipment,  each  of 
the  data  types  was  subjected  to  an  empirical  demonstration  of  its 
accuracy  in  permitting  the  isolation  of  equipment  problems. 

The  equipment  problems  against  which  the  documentation  types  were 
evaluated  were  selected  after  an  intensive  review  of  worldwide  Air  Force 
maintenance  actions  on  the  AN/APN-147  and  AN/ASN-35  systems  for  calendar 
year  1974.  The  data  contained  in  these  reports  were  summarized  to 
provide  identification  of  maintenance  areas  which  appeared  to  be  logical 
and  representative  candidates  for  development  of  troubleshooting  test 
problems.  In  an  attempt  to  provide  field  verification  of  the  correct- 
ness of  the  data  summarizations,  the  results  were  discussed  in  detail 
with  maintenance  personnel  at  Charleston  AFB,  South  Carolina.  (This 
base  was  not  included  in  the  field  evaluation  effort.)  As  a result  of 
the  Charleston  AFB  interviews,  detailed  information,  not  available  from 
the  AFLC  reports,  was  obtained  on  the  specific  parts  or  components  which 
were  contributing  most  to  the  maintenance  problems  of  the  AN/APN-147  and 
the  AN/ASN-35.  This  information  was  considered  in  selecting  the  specific 
troubleshooting  test  problems.  A total  of  30  malfunotions  was  selected 
and  located  on  an  applicable  system  schematic  to  verify  that  the  problems 
selected  were  satisfactorily  distributed  throughout  the  system. 

Each  of  the  candidate  problems  was  investigated  in  working  equip- 
ment to  catalog  the  operational  symptoms  associated  with  the  failed 
equipment  part  or  component.  Once  the  known  effects  of  the  fault  had 
been  identified,  a systematic  evaluation  of  the  accuracy  of  each  of  the 
technical  documents  in  supporting  the  isolation  of  the  faults  was 
undertaken.  This  evaluation  was  based  on  actual  troubleshooting  perform- 
ance on  powered  equipment  using  each  of  the  document  types  as  the 
guiding  document  for  fault  isolation.  The  approach  just  described  made 
use  of  the  same  (primary  and  test)  equipment  and  aids  as  used  in  the 
field  evaluation  portion  of  the  experiment. 

Certain  problems  had  to  be  substituted  to  assure  compatibility  with 
the  degree  of  simulation  fidelity  employed  in  the  Air  Force  simulations 
of  a C-141A  cockpit  and  intermediate  (shop)  level  maintenance  benches. 

As  an  example,  the  technical  documentation  guiding  isolation  of  certain 
of  the  faults  originally  selected  directed  the  troubleshooter  to  the 
Horizontal  Situation  Indicator  (HSI).  The  HSI  was  represented  by  a 
nonfunctioning  simulation  in  the  Air  Force  trailer;  thus,  the  particular 
fault  could  not  be  resolved  by  the  technical  documentation/equipment 
simulation  combination.  In  an  instance  such  as  this,  a new  candidate 
fault  was  selected  and  the  ability  of  the  types  of  technical  documentation 
to  find  it  was  verified.  Any  editorial  or  content  problem  with  the 
candidate  types  of  technical  documentation  discovered  as  a result  of 
following  this  process  was  reported  to  the  Air  Force  technical  monitor 
for  correction. 


10 


Eighteen  airmen  graduates  of  Keesler  Technical  Training  Center 
(KTTC)  Course  No.  3ABR32834  served  as  one  of  the  three  subject  groups. 
Subjects  were  first-enlistment  airmen  who  had  completed  basic  training 
and  who  went  directly  into  technical  training  at  KTTC.  The  airmen  were 
retained  at  Keesler  for  a two-week  period  beyond  course  graduation  and 
entered  as  subjects  in  this  experiment. 

The  two  remaining  subject  groups  in  the  design  were  obtained  from 
Air  Force  enlisted  personnel  in  the  328X4  AFSC,  Avionics  Inertial  and 
Radar  Navigational  Systems  Specialist,  assigned  to  operational  units  of 
the  Military  Airlift  Command  (MAC)  and  performing  maintenance  on  the 
AN/ASN-35  and  AN/APN-147  systems  in  C-141A  aircraft. 

These  two  groups  were  classified  on  the  basis  of  length  of  experi- 
ence with  the  C-141A  equipment.  Each  group  had  a size  of  n«18.  One 
group  consisted  of  airmen  with  six  months  or  less  equipment  experience 
on  the  system;  the  second  group,  airmen  with  more  than  six  months 
experience  on  the  system.  Three  MAC  bases  were  visited  to  obtain  the 
required  number  of  subjects.  Each  airman  selected  as  a subject  was  made 
available  for  a two-week  period.  During  this  period,  the  airmen  con- 
tinued to  work  their  normal  shifts  and  served  as  subjects  for  a four- 
hour  period  out  of  each  eight-hour  shift. 

An  Air  Force  trailer  configured  to  include  AN/ASN-35  (computer)  and 
AN/APN-147  (radar)  bench  mock-ups  and  a mock-up  of  the  C-141A  cockpit 
comprised  the  main  experimental  apparatus.  In  addition,  a number  of 
items  of  test  equipment  were  available  for  use  with  the  prime  equipment. 

Equipment  components  had  faults  introduced  in  a manner  which  would 
produce  15  equipment  problems  with  known  symptoms  and  known  overall 
effects  on  equipment  operating  characteristics.  These  failed  components 
were  provided  for  the  experimenters  to  insert  in  the  equipment  following 
the  order  required  by  the  experimental  procedure. 


METHODOLOGY  FOR  FIELD  EVALUATION 


In  the  field  data  collection  phase  of  the  effort,  selected  subjects 
were:  evaluated  for  proficiency  with  13  separate  items  of  test  equip- 

ment; provided  with  remedial  training  on  those  items  of  test  equipment 
where  proficiency  was  below  the  criterion  established;  trained  to 
criterion  in  the  use  of  FPTA  and  LTTA;  and  evaluated  for  their  ability 
to  successfully  isolate  predetermined  faults,  which  had  been  inserted 
into  equipment,  using  the  types  of  technical  data  under  evaluation. 


SUBJECTS 

An  evaluation  sample  of  54  Air  Force  enlisted  personnel  in  the 
328X4  AFSC  (Avionics  Inertial  and  Radar  Navigational  S/stems  Specialist) 
was  studied.  Of  this  total  of  54  subjects,  18  were  3-level  airmen  who 
were  recent  graduates  of  the  Keesler  Technical  Training  Center  Course 
3ABR32834.  The  remaining  36  subjects  were  obtained  from  Air  Force 
operational  units.  Eighteen  of  these  36  were  airmen  who  had  either  been 
awarded  the  5-level  of  the  AFSC  within  the  last  six  months  or  for  whom 
award  was  imminent.  The  remaining  18  subjects  were  5-level  airmen  with 
more  than  six  months  experience  on  the  system.  A determination  was  made 
that  it  would  not  be  appropriate  to  test  the  Keesler  group  on  -technical 
orders.  This  decision  made  it  necessary  to  use  two  different,  but 
similar,  experimental  designs.  These  designs  are  described  in  detail  in 
the  following  sections. 


TESTING  LOCATIONS 

An  Air  Force  trailer,  configured  to  include  AN/ASN-35  and  AN/APN-147 
bench  mock-ups  and  a mock-up  of  the  C-141A  cockpit,  was  taken  to  four 
separate  Air  Force  locations  (Keesler  Air  Force  Base,  Mississippi; 

McChord  Air  Force  Base,  Washington;  Travis  Air  Force  Base,  California; 
and  McGuire  Air  Force  Base,  New  Jersey).  The  trailer  served  as  the 
experimental  setting  for  the  controlled  experimental  evaluation. 

It  was  recognized  that  it  would  be  extremely  difficult  to  obtain 
adequate  access  to  C-141  aircraft  for  use  with  the  organizational  level 
tests.  Thus,  the  mock-up  of  the  C-141  cockpit  and  the  avionics  equip- 
ment bays  of  the  C-141  was  developed  and  Installed  in  the  test  van.  The 
cockpit  mock-up  provided  a means  of  simulating  the  flightline  trouble- 
shooting tasks  without  requiring  access  to  an  aircraft.  It  consisted  of 
mock-ups  of  all  instriment  panels  in  the  cockpit.  All  AN/APN-147  and 
AN/ASN-35  components  were  "live"  and  functioned  in  the  same  manner  as 
the  actual  cockpit.  All  other  components  on  the  Instrument  panels  were 
represented  by  photographs.  Figure  1 is  a photograph  of  the  exterior  of 
the  van.  Figure  2 is  a view  of  the  cockpit  mock-up  area  inside  the  van. 
Figures  3 and  4 are  views  of  the  bench  mock-ups  contained  in  the  van. 
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Figure  3.  View  of  the  Radar  Maintenance  Bench 


Figure  4.  View  of  the  Computer  Maintenance  Bench 


ITEMS  CONSIDERED 


Procedures  were  considered  for  handling  the  following  items  of 

concern: 


Subjects 

Graduating  classes 

Order  of  technical  data  training 

Equipment  faults 

Test  equipment  proficiency  testing 
Remedial  training  on  test  equipment 
Data  collection  on  ancillary  factors 

Subject  availability  was  different  at  the  operational  locations 
than  at  KTTC.  For  this  reason,  the  manner  of  consideration  of  each  of 
the  above  variables  has  been  outlined  separately  for  the  two  experi- 
mental situations  encountered. 


KEESLER  TECHNICAL  TRAINING  CENTER 

At  this  location,  airmen  were  available  on  a full-time  basis  for 
two  weeks  following  their  graduation  from  Course  3ABR32834.  Classes 
graduate  at  biweekly  intervals.  Graduation  dates  and  sizes  of  the 
classes  used  were  as  follows:  7 October,  7 graduates;  21  October,  9 
graduates;  4 November,  9 graduates. 

At  Keesler,  each  class  was  treated  as  a separate  unit,  and  entered 
into  the  study  as  described  in  the  following  sections.  Each  graduate 
was  assigned  to  an  Initial  evaluation  group  by  means  of  a random  selec- 
tion process. 

Test  Equipment  Proficiency  Testing  and  Remedial  Training 

Thirteen  items  of  test  equipment  were  required  in  this  evaluation 
for  performing  troubleshooting  procedures.  Each  of  the  performance 
tests  was  available  to  evaluate  the  level  of  proficiency  of  each  tech- 
nician in  using  the  13  items  of  equipment  (Figure  5) . Discussion  with 
Air  Force  personnel  resulted  in  the  consensus  that  the  recent  Keesler 
graduates  would  not  be  proficient  in  the  test  equipment  items  called  out 
in  Figure  1.  Due  to  the  short  period  of  time  the  evaluation  team  had 
access  to  each  class  of  graduates.  It  was  decided  that  more  effective 
utilization  of  time  would  be  accomplished  through  bypassing  the  profi- 
ciency tests  and  subjecting  each  graduate  to  the  complete  set  of  remedial 
training  courses.  This  procedure  was  followed  with  each  of  the  three 
classes  at  Keesler. 

An  example  of  the  schedule  followed  in  conducting  the  remedial 
training  on  the  items  of  test  equipment  is  depicted  in  Figure  6.  In 
this  figure,  the  number  appearing  in  a block  in  any  column  can  be  used 


1 

410B  VTVM  Vacuum  Tube  Voltmeter 

2 

260  VOM  Volt  Ohmmeter 

3 

545B  Oscilloscope 

4 

5245L  Electronic  Counter  and  Plug-Ins 

5 

TV2  Tube  Tester 

6 

TS-148  Radar  Test  Set 

Not 

Used 

7 

803  Differential  VTVM 

Not 

Used 

8 

200CD  Audio  Oscillator 

9 

URM-25D  Signal  Generator 

10 

TS-1100/U  Test  Set,  Transistor 

Not 

Used 

11 

CMA-544  Doppler  Simulator 

12 

CMA-546  Doppler  Generator 

13 

3322-900  Computer  Cards  Tester 

14 

3322-902  Computer  Drives  Tester 

15 

3322-905  Navigational  Tester 

16 

3322-901  Computer  Relay  Chassis  Tester 

Not 

Used 

17 

SG299B/U  Signal  Generator 

18 

Kay  Sweep  Generator 

Not 

Used 

Figure  5.  List  of  Remedial  Training  Courses  for 
18  Items  of  Test  Equipment 


ure  6.  Test  Equipment  Remedial  Training  Schedule  for  a Class  Size 


to  identify  the  item  of  test  equipment  for  which  remedial  training  was 
to  be  conducted  in  that  time  period.  In  addition,  a training  program 
was  included  in  this  schedule  to  familiarize  the  graduates  with  the 
circuit  breaker  panel  included  as  part  of  the  C-141A  mock-up  in  the  Air 
Force  trailer. 


EQUIPMENT  FAULTS 

A total  of  15  equipment  faults  were  selected  for  use  as  test 
problems  in  the  experimental  evaluation  (Figure  7).  These  faults  were 
organized  so  that  four  were  representative  of  organizational  level 
maintenance  troubleshooting  tasks  on  the  AN/ASN-35  and  AN/APN-147 
equipment.  The  remaining  11  faults  were  representative  of  intermediat< 
level  maintenance  troubleshooting  tasks. 

At  the  organizational  level  of  maintenance,  an  equal  number  of 
computer  (ASN-35)  and  radar  (APN-147)  faults  were  Included.  Thus,  the 
four  organizational  level  faults  consisted  of  two  computer  malfunctions 
and  two  radar  malfunctions.  The  11  intermediate  level  faults  included 
five  computer  and  six  radar  malfunctions. 

One  of  the  intermediate  level  faults  was  not  used  with  the  Keesler 
graduates.  This  fault  (Number  9 in  Figure  7)  was  excluded  to  balance 
the  intermediate  level  problems  at  Keesler  to  five  computer  and  five 
radar  faults. 

An  attempt  was  made  with  the  organizational  and  intermediate  level 
faults  to  categorize  them  on  an  additional  dimension:  levels  of  trouble- 

shooting. Three  levels  of  troubleshooting  were  used  for  this  purpose — 
troubleshoot  to  unit,  to  subassembly,  and  to  part. 

Both  the  organizational  and  intermediate  level  faults  were  counter- 
balanced in  presentation  order  to  subjects,  and  each  fault  was  isolated 
an  equal  number  of  times  with  LTTA  and  with  FPTA  documentation. 


ORDER  OF  TECHNICAL  DATA  TRAINING 

Reference  is  made  to  the  order  of  counterbalancing  scheme  diagrammed 
in  Figure  8.  The  type  of  technical  data  training  received  first  by  the 
subjects  was  determined  by  random  assignment  of  a subject  to  one  of  the 
18  counterbalanced  orders  of  technical  data  for  problem  solving.  In 
developing  these  counterbalancing  orders,  a systematic  counterbalancing 
scheme  was  used  for  both  the  organizational  and  the  intermediate  level 
problems  with  the  check  that  each  problem  was  tested  nine  times  with 
FPTA  and  nine  times  with  LTTA. 

As  can  be  seen  by  consulting  Figure  8,  the  first  seven  sequences 
started  with  an  FPTA  in  four  instances,  and  with  an  LTTA  in  three 
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*Problem  9 not  used  with  Kessler  AFB  subjects. 

Figure  7.  Identification  of  Problems  used  in  Field  Evaluation  of  Three  Types  of 
Technical  Data,  with  Expected  Time  Required  to  Troubleshoot 


Figure  8.  Order  of  Counterbalancing  of  Problems  to  Technical  Data  Type 
for  Keealer  AFB  Subjects 
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instances.  In  Figure  9,  the  18  sequences  have  been  reordered  to  group 
into  type  of  technical  documentation  first  used.  The  sequences  were 
also  partialed  into  class  size  groups  matching  those  encountered  at 
Keesler.  As  mentioned  earlier,  the  first  Keesler  class  had  7 graduates, 
the  second  class  had  9,  and  the  third  9 also.  For  the  third  class,  only 
the  two  sequences  required  to  complete  the  planned  sample  size  are 
depicted.  The  7 graduates  remaining  in  the  third  class  were  given  a 
repeat  of  the  sequence  used  for  the  first  graduating  class. 

The  pairing  of  subject  to  sequence  is  shown  in  Figure  9.  This 
pairing  was  accomplished  within  each  Keesler  graduating  class  by  means 
of  a random  selection  process. 

For  the  first  two  subjects,  the  sequence  of  intermediate  level 
problems  evaluated  was  arbitrarily  assigned.  These  two  sequences  were 
than  systematically  counterbalanced  until  the  presentations  were  exhausted. 
A new  sequence  was  generated  by  a random  selection  process,  using  the 
restriction  that  no  sequential  arrangement  of  two  or  more  problems  could 
be  repeated.  This  new  sequence  was  systematically  counterbalanced  until 
its  presentations  were  exhausted.  The  procedure  was  then  repeated  to 
secure  the  necessary  number  of  different  orders  of  presentation. 

Since  both  the  radar  and  computer  bench  mock-ups  in  the  Air  Force 
trailer  were  used  simultaneously,  a rigid  computer-radar  problem  pairing 
had  to  be  observed.  The  restriction  mentioned  above  had  to  be  violated 
twice  to  secure  the  required  problem  pairing  on  the  intermediate  problems. 

A separate  counterbalancing  process  was  used  for  the  organizational 
level  problem  sequence. 

Shown  in  Figures  10,  'll,  and  12  are  the  complete  training  and 
testing  schedules  used  at  Keesler  with  each  of  the  three  classes. 

Subjects 

The  subject  pool  at  Keesler  has  been  described  earlier.  Basically, 

25  subjects  were  available  from  which  the  sample  of  18  was  drawn.  All 
graduating  classes  were  used  in  toto , providing  25  subjects  on  which 
cosiplete  data  were  collected.  The  decision  was  made  to  omit  the 
7 October  class  from  the  analysis. 

To  facilitate  subject  reporting  over  the  two-week  period  of 
testing,  a separate  schedule  was  prepared  for  each  subject  indicating 
where  the  subject  would  be  utilized  (and  where  he  was  to  report)  for 
each  one-half  day  of  the  evaluation  period.  An  example  of  these 
schedules  for  one  airman  is  presented  in  Table  1. 

Comparable  information  was  provided  in  schedules  for  each  of  the 
four  test  administrators  (see  Table  2).  Additionally,  in  this  schedule, 
an  identification  of  the  problem  by  number  and  the  technical  data  format 
to  be  used  with  the  problem  was  Included. 
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Training  and  Testing  Schedule  for  Keesler  AFB  Students,  Class  Size 


Figure  11.  Training  and  Testing  Schedule  for  Keesler  AFB  Students,  Class  Size 


Figure  12.  Training  and  Testing  Schedule  for  Keesler  AFB  Students;  Class  Size  n - 9,  where  S17  and  Si  a 
Complete  Original  Sequence,  S19  Through  25  Start  Repeat  of  Sequence 


TABLE  1.  EXAMPLE  OF  INDIVIDUAL  TECHNICIAN  SCHEDULE  SHEET 
FOR  KEESLER  AFB  TECHNICAL  DATA  EVALUATION 


Date 

Time 

Location 

Wed  8 October 

AM 

Classroom 

PM 

Classroom 

Thurs  9 October 

AM 

Classroom 

PM 

Classroom  f 

Fri  10  October 

AM 

Classroom 

PM 

t 

Tues  14  October 

AM 

t 

PM 

Radar  Bench/Trailer  , 

Weds  15  October 

AM 

Radar  Bench/Trailer 

PM 

t 

Thurs  16  October 

AM 

Cockpit/Trailer 

PM 

Classroom 

Fri  17  October 

AM 

Radar  Bench/Trailer 

PM 

t 

Mon  20  October 

AM 

t 

PM 

Cockpit/Trailer 

Tues  21  October 

AM 

Radar  Bench/Trailer 

PM 

t 

*AM  - 0730  to  1130;  PM  - 1230  to  1630 
tAs  notified  by  experimental  team. 
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TABLE  2.  EXAMPLE  OF  SCHEDULING  OF  EXPERIMENTERS  AND  SUBJECTS  FOR 
DAYS  3 THROUGH  9,  7 OCTOBER  GRADUATING  CUSS 


COCKPIT 

(Experimenter  Name) 

Problems 


Day  3-10  October 


(PM) 

A1C  (Subject 

name) , S-5 

L-3,1 

Day  4-14 

October 

(AM) 

A1C  ( 

),  S-6 

L-2,1 

(PM) 

A1C  ( 

),  S-4 

F-2,3 

Day  5-15 

October 

(AM) 

AMN  ( 

),  S-3 

F-1,4 

(PM) 

AflN  ( 

),  S-7 

L-3,4 
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Data  Collection  on  Ancillary  Factors 


During  the  troubleshooting  evaluation  portion  of  the  experiment, 
each  subject  was  scheduled  for  troubleshooting  for  only  one-half  of  each 
day  (refer  to  Table  1).  The  half-day  so  committed  was  alternated  (refer 
o igures  10,  11,  and  12)  in  an  attempt  to  remove  any  order  of  time- 
of-day  effect  on  a particular  problem,  technical  data  type,  or  subject. 

The  one-half  day  not  involved  with  troubleshooting  was  used  to 
collect  ancillary  data.  Factors  on  which  data  were  being  collected 
included  background,  personality,  attitude  and  aptitude.  The  instruments 
used  in  this  process  included  the  USAFI  reading  level  test.  No.  3; 
attitude  and  electronic  background  questionnaires  developed  for  this 
experimental  evaluation  (Appendices  A,  B,  C,  and  D);  and  a 376  item 
Adjustment  Prediction  Inventory  developed  by  the  author  (Flyer  and 
Potter,  1959).  Imbedded  in  this  inventory  is  the  L.V.  Gordon  Profile 
(Gordon,  1953)  and  Personal  Inventory  (Gordon,  1956). 

E®rlier  dl8Cussion  indicated  the  decision  to  forego  initial  testing 
with  the  test  equipment  proficiency  tests  in  the  interest  of  time 
conservation.  Since  these  instruments  had  no  field  validation,  it  was 
decided  to  administer,  to  the  extent  equipment  availability  permitted, 
these  tests  during  the  nontroubleshooting  period  of  the  day.  Using  this 
procedure,  test  performance  data  were  collected  on  the  proficiency  tests 
covering  the  410B  VTVM  (Item  1 in  Figure  1),  Simpson  260  VOM  (Item  2), 
545B  Oscilloscope  (Item  3),  and  TV  Tube  Tester  (Item  5). 

Procedure 


First  access  to  each  graduating  class  was  for  a one-half  hour  period 
on  the  day  of  graduation.  During  this  period,  the  members  of  the  evalua- 
tion team  were  introduced;  the  purpose  of  the  evaluation  was  described  to 
the  class;  each  class  member  was  given  his  initial  assignment  for  the 
following  day;  a tour  of  the  trailer  and  classroom  evaluation  areas 
was  conducted;  and  routine  administrative  announcements  (e.g.,  messing, 
duty  hours,  and  reporting  chain  of  command  for  the  two-week  period)  were 
made. 


Day  one  and  two  of  the  class  availability  were  devoted  to  test 
equipment  (remedial)  training  following  the  schedules  shown  in  Figure  2. 
The  morning  of  day  three  was  devoted  to  FPTA  or  LTTA  training,  depending 
on  the  group  to  which  the  airman  was  assigned.  The  afternoon  of  the 
third  day  was  devoted  either  to  troubleshooting  of  problems  or  to 
ancillary  factor  data  collection.  Again,  the  activity  in  which  a class 
member  was  engaged  depended  upon  the  group  to  which  he  was  assigned. 

Days  four  and  five  and  the  morning  of  day  six  were  devoted  to  trouble- 
shooting and  ancillary  data  collection  following  schedules  reflected  in 
Table  1 and  Figures  10,  11,  and  12. 

After  he  had  completed  all  problems  with  the  first  type  of  data, 
each  airman  was  asked  to  complete  an  opinion  questionnaire  on  the  data 
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format  he  had  just  used.  After  completing  this  form,  he  was  entered  (on 
the  afternoon  of  day  six)  into  cross-training  on  the  use  of  the  second 
experimental  technical  documentation  format.  Days  seven,  eight,  and 
nine  repeat  the  troubleshooting/ancillary  data  collection  cycle  described 
for  days  four,  five  and  six. 

The  evaluation  cycle  for  the  class  was  completed  at  the  close  of 
day  nine  (day  ten  for  the  third  graduating  class)  and  the  class  was 
returned  to  regular  duty  (delay  enroute  to  first  operational  assignment 
as  an  avionics  inertial  and  radar  navigation  systems  specialist). 

At  the  completion  of  day  nine  troubleshooting  problems,  the  airman 
completed  an  opinion  questionnaire  on  the  data  format  he  had  just  used 
and  then  a second  opinion  questionnaire  which  asked  him  to  compare  the 
two  formats.  Copies  of  the  questionnaires  are  given  in  Appendices  D 
and  E. 

Testing  Procedure 

For  each  unique  troubleshooting  problem,  the  test  administrator 
prepared  the  equipment  by  completing  the  "pretest  setup"  section  of  the 
problem  contained  in  the  Test  Administrator's  Guide  (AFHRL-TR-76-74(III)) . 
In  addition,  the  presence  of  the  support  material  requirements  (listed 
in  Section  IV  of  each  problem)  was  verified.  At  this  point,  the  subject 
was  brought  into  the  troubleshooting  situation.  Following  the  instruc- 
tions section  (Section  V)  of  the  guide,  the  subject  was  given  the 
information  and  instructions  sheet  (Appendix  F)  and  asked  to  read  it. 

He  was  then  given  the  specific  instructions  sheet  and  the  work  order 
which  described  the  problem  he  was  to  troubleshoot  (Appendix  G) . 

(Refer  to  DD  Form  1473,  Block  18,  AFHRL-TR-76-74(II) .) 

The  troubleshooting  evaluation  was  conducted  on  a one-on-one  basis 
(one  test  administrator  to  one  subject).  The  test  administrator  recorded 
data  on  student  performance  using  such  measures  as  time,  errors,  parts 
consumed,  successful  completion  of  each  section,  branching  followed  (and 
correctness)  and  success  in  troubleshooting.  (See  data  recording  forms 
included  in  Test  Administrator's  Guide.)  The  test  administrator  was 
instructed  to  answer  no  procedural  questions  while  the  test  was  ongoing. 
(The  adequacy  of  the  document  to  lead  the  student  to  the  correct  equip- 
ment malfunction  was  the  basic  item  being  evaluated.) 

At  no  point  in  the  troubleshooting  process  with  either  the  Keesler 
or  the  Military  Airlift  Command  subjects  was  the  technician  permitted  to 
effect  repairs  of  the  equipment.  In  an  operational  situation,  a trouble- 
shooting philosophy  adopted  frequently  includes  removal  and  replacement 
of  suspected  parts  and  components,  in  effect,  a trial  and  error  approach 
to  fault  isolation.  Following  such  an  approach  to  troubleshooting  with 
the  experimental  equipment  (the  cockpit,  and  AN/ASN-35  and  AN/APN-147 
bench  mock-ups)  would  have  exposed  this  equipment  to  an  Inordinate 
amount  of  abuse.  The  result  could  well  have  placed  the  evaluation  in 
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jeopardy  either  through  causing  total  equipment  failure,  or  worse, 
by  introducing  insidious  malfunctions  which  would  permit  the  equipment 
to  continue  operating  but  which  would  produce  fallacious  equipment 
operating  symptoms  and  signals.  Thus,  in  the  test  situation  when  a 
technician  indicated  that  he  would  remove  and  replace  a specific  com- 
ponent which  was  good,  he  was  told  to  assume  that  he  had  done  so  and  to 
continue  as  if  he  had  replaced  the  part.  Standard  times  were  derived 
for  removal  and  replacement  of  the  components  and  parts  identified  by 
technicians  in  the  evaluation  and  these  times  were  added  to  subject  time 
to  obtain  a total  elapsed  time  to  fault  isolation. 


■ 
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MILITARY  AIRLIFT  COMMAND  (MAC)  OPERATIONAL  SITES 

The  two  remaining  sample  groups  were  obtained  from  Air  Force 
enlisted  personnel  assigned  in  the  328X4  AFSC,  who  served  as  Avionics 
Inertial  and  Radar  Navigational  Systems  Specialists,  in  operational 
units  in  MAC  and  performed  maintenance  on  the  AN/ASN-35  and  AN/APN-147 
systems  of  the  C-141  aircraft. 

Each  of  the  two  remaining  sample  groups  had  a size  of  n-18.  The 
first  of  these  groups  was  composed  of  airmen  with  AFSC  328X4  who  had 
less  than  six  months  work  experience  on  the  systems;  the  second  group, 
airmen  in  AFSC  328X4  with  six  months  or  more  work  experience  on  the 
systems. 

Testing  Locations  and  Subject  Availability 

The  operational  locations  used  for  obtaining  subjects  were  McChord, 
Travis,  and  McGuire  Air  Force  Bases. 

All  MAC  sites  visited  were  operating  on  a three-shift  day.  Subjects 
from  each  of  the  three  shifts  were  required  to  obtain  the  required 
sample  sizes.  Each  individual  was  made  available  for  a four-hour  block 
of  time  each  day  that  he  participated  in  the  evaluation.  Each  individual 
participated  each  day  for  four  hours  for  a two-week  period.  Three 
shifts  of  Air  Force  maintenance  personnel  were  covered  with  a two-shift 
operation  on  the  part  of  the  test  administrators. 

Test  Equipment  Testing  and  Remedial  Training 

The  proficiency  tests  evaluating  a technician's  ability  to  use 
specific  items  of  test  equipment  were  administered  on  a selective  basis. 
It  was  believed  that  a technician  with  field  experience  had  a realistic 
assessment  of  his  familiarity  with  a specific  item  of  test  equipment. 

For  this  reason,  it  was  believed  that  presenting  a technician  with  a 
list  of  test  equipment  (see  Figure  5)  and  asking  him  to  rank  his  ability 
to  use  each  of  the  items  would  reveal  those  on  which  his  proficiency 
would  be  verified.  Except  for  items  requiring  that  all  subjects  be 
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placed  into  training  to  ensure  greater  familiarity,  proficiency  tests 
were  administered  only  when  the  subject  technician  indicated  a question- 
able proficiency. 

Equipment  Faults 

The  MAC  subjects  were  exposed  to  the  full  set  of  15  equipment 
faults  (see  Figure  7) . Four  of  the  faults  were  representative  of 
organizational  level  maintenance  troubleshooting  tasks  on  the  AN/ASN-35 
and  AN/APN-147  equipment.  The  remaining  11  faults  were  representative 
of  intermediate  level  maintenance  troubleshooting  tasks. 

At  the  organizational  level  of  maintenance,  an  equal  number  of 
computer  (ASN-3S)  and  radar  (APN-147)  faults  were  included.  Thus,  the 
four  organizational  level  faults  consisted  of  two  computer  malfunctions 
and  two  radar  malfunctions.  The  11  Intermediate  level  faults  included 
five  computer  and  six  radar  malfunctions. 

An  attempt  was  made  with  the  organizational  and  intermediate  level 
faults  to  categorize  them  on  an  additional  dimension:  levels  of  trouble- 

shooting. Three  levels  of  troubleshooting  were  used  for  this  purpose: 
troubleshoot  to  unit,  troubleshoot  to  subassembly,  and  troubleshoot  to 
part. 


Both  the  organizational  and  intermediate  level  faults  were  counter- 
balanced in  presentation  order  to  subjects,  and  each  fault  was  isolated 
an  equal  number  of  times  with  TO,  LTTA,  and  FPTA  documentation. 

Order  of  Technical  Data  Training 

Reference  is  made  to  the  order  of  counterbalancing  scheme  diagrammed 
in  Figure  13.  In  this  portion  of  the  evaluation,  subjects  were  exposed 
to  all  three  formats  of  technical  data:  TOs,  FPTAs,  and  LTTAs.  The 

order  in  which  the  documentation  types  were  assigned  to  problems  was 
determined  by  random  selection.  Thus,  sequence  (TO,  FPTA,  LTTA)  was 
assigned  for  the  first  three  problems  subject  one  encountered,  in  both 
organizational  and  intermediate  troubleshooting.  This  sequence  was  then 
repeated  to  fill  the  4 * 18  matrix  of  cells  for  the  organizational 
level  problem  presentation  and  the  11  x 18  matrix  for  the  intermediate 
problem  presentation.  Thus,  presentation  order  was  systematically 
counterbalanced  with  problems  and  each  data  type  was  used  an  equal 
number  of  times  with  each  p'  Mem. 

Each  subject  with  six  months  or  less  experience  on  the  system  was 
paired  with  a number  from  1 to  18  using  a random  assignment  process. 

This  assignment  process  determined  the  particular  sequence  in  which  the 
problems  were  presented  to  the  subject.  Further,  the  18  subjects  were 
separated  into  three  equal  size  groups  (subjects  1 through  6 [Figure 
14],  subjects  7 through  12  [Figure  15],  subjects  13  through  18  [Figure 
16]).  The  first  of  these  groups  was  assigned  a sequence  of  document 
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ORGANIZATIONAL 


. INTERMEDIATE 


Figure  13.  Order  of  Counterbalancing  of  Problems  and  Data  Format  Subjects  for 

Military  Airlift  Command  Where  Subjects  Fall  in  the  Group  of  5-Level 
Technicians  with  6 Months  or  Less  Experience  in  the  AFSC 
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Figure  14.  Presentation  Sequence  for  Problems  by  Subjects  by  Type 
of  Technical  Documentation  for  Military  Airlift  Command 
Subjects  Where  Subjects  are  5-Level  with  6 Months  or 
Less  Experience  at  That  Level 
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Figure  15.  Presentation  Sequence  for  Problems  by  Subjects  by  Type 
of  Technical  Documentation  for  Military  Airlift  Command 
Subjects  Where  Subjects  are  5-Level  with  6 Months  or 
Less  Experience  at  That  Level 
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Figure  16.  Presentation  Sequence  for  Problems  by  Subjects  by  Type 
of  Technical  Documentation  for  Military  Airlift  Comnand 
Subjects  Where  Subjects  are  5-Level  with  6 Months  or 
Less  Experience  at  That  Level 


type.  The  subjects  In  Group  1 (Figure  14)  solved  problems  first  with 
FPTA  documentation;  second  with  LTTA;  and  last,  with  TO.  Order  of 
presentation  was  counterbalanced  for  the  other  two  groups  with  the 
following  results:  Group — 2 LTTA,  TO,  FPTA;  Group  3 — TO,  FPTA,  LTTA. 

The  sequence  of  problems  and  orders  of  data  format  presentation  which 
resulted  for  the  subjects  having  six  months  or  less  experience  as 
5-level  technicians  are  presented  in  Figures  13,  14,  15,  and  16. 

Comparable  schedules  for  the  sample  group  (size  n-18)  composed  of 
5-level  technicians  with  more  than  six  months  experience  are  presented 
in  Figures  17,  18,  19,  and  20. 

To  assemble  the  sequences  and  orders  depicted  in  Figures  17,  18, 

19,  and  20  a new  group  of  18  numbers  (19  through  36)  was  randomly 
assigned  to  the  problem  sequences  appearing  in  Figure  13,  resulting  in 
the  schedule  appearing  as  Figure  17.  The  content  of  Figures  18,  19,  and 
20  was  generated  from  the  data  in  Figure  17  following  the  procedure 
described  previously. 

Subjects 

The  characteristics  of  the  subject  groups  obtained  from  MAC  opera- 
tional sites  have  been  discussed  in  earlier  sections.  An  example  of  the 
schedules  used  at  each  of  the  MAC  sites  is  presented  in  Figures  21 
through  23. 

Data  Collection  on  Ancillary  Factors 

With  the  MAC  sample  groups,  with  the  exception  of  the  United  States 
Air  Force  Institute  (USAFI)  reading  score,  identical  ancillary  data  to 
that  collected  with  the  Keesler  subjects  were  obtained.  Ancillary  data 
were  obtained  at  any  time  the  subject  was  available  and  was  not  required 
for  primary  data  collection  activities. 

Procedure 


The  procedure  followed  at  each  MAC  base  was  dependent  upon  the 
agreements  reached  with  base  personnel.  As  the  base  requirements  were 
revealed,  a procedure  was  established  so  that  a standard  approach  to 
each  subject  on  that  base  was  maintained.  To  the  extent  possible,  the 
procedure  at  each  MAC  base  was  the  same  and  conformed  as  nearly  to  the 
Keesler  subject  procedure  as  was  practical. 

Variations  in  procedures  from  base  to  base  were  primarily  admin- 
istrative with  respect  to  access  to  technicians  and  physical  facilities. 
In  all  cases,  once  the  subject  was  placed  in  the  troubleshooting  problem, 
he  was  treated  in  an  Identical  manner  to  Keesler  subjects  and  the  same 
measures  were  obtained  on  the  subject. 
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Figure  17.  Order  of  Counterbalancing  of  Problems  and  Data  Format  by  Subjects  for 
Military  Airlift  Command  Where  Subjects  Fall  In  the  Group  of  5-Level 
Technicians  with  More  Than  6 Months  Experience  In  the  AFSC 
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Figure  18.  Presentation  Sequence  for  Probleas  by  Subjects  by  Type  of 
Technical  Documentation  for  Military  Airlift  Command 
Subjects  Where  Subjects  are  5-Level  with  More  than  6 Months 
Experience  at  That  Level 
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Figure  19.  Presentation  Sequence  for  Problems  by  Subjects  by  Type 
of  Technical  Documentation  for  Military  Airlift  Command 
Subjects  Where  Subjects  are  5-I.evel  with  More  than  6 
Months  Experience  at  That  Level 
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Figure  21.  Schedule  of  Travis  AFB  Subjects,  by  Subject  Number  and 
Time  Available,  for  the  First  Two  Weeks  of  Evaluation 
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Figure  22.  Schedule  of  Travis  AFB  Subjects,  by  Subject  Number  and 
Time  Available,  for  the  Second  Two  Weeks  of  Evaluation 
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Subject 

■379 

Problems 

■379 

Problems 

■379 

Problems 

No. 

32 

TO 

4,1,10,13,7 

FPTA 

8,14,11,5,2 

LTTA 

3,9,12,15,6 

19 

FPTA 

7,10,13,1,4 

LTTA 

2,5,8,11,14 

TO 

6,9,12,15,3 

27 

LTTA 

13,7,10,1,4 

TO 

2,11,14,8,5 

FPTA 

12,15,6,9,3 

20 

FPTA 

3,9,12,15,6 

LTTA 

10,13,7,4,1 

TO 

2,5,8,11,14 

10 

LTTA 

3,15,6,9,12 

TO 

7,10,13,4,1 

FPTA 

2,14,8,5,11 

1 

FPTA 

5,8,11,14,2 

LTTA 

3,6,9,12,15 

TO 

7,10,13,1,4 

15 

TO  * 

12,15,6,9,3 

FPTA 

1,4,13,7,10 

LTTA 

14,11,5,8,2 

4 

FPTA 

2,14,11,8,5 

LTTA 

15,6,9,12,3 

TO 

4,1,7,10,13 

SECOND  2 WEEKS 


Subject 


Problems 


TO  13,7,10,1,4 
LTTA  3,15,6,9,12 
LTTA  5,8,11,14,2 
TO  6,9,12,15,3 


LTTA  13,7,10,1,4  TO 

LTTA  4,1,13,7,10  TO 


FPTA 

TO 

TO 

FPTA 

FPTA 

TO 


Problems 

2,14,8,5, 

11 

7,10,13,4 

,1 

3,6,9,12, 

15 

1*4,7 ,io , 

13 

10,13,7,4 

,1 

2,5,8,11, 

14 

14,11,8,5 

.2 

7,10,13,4 

,1 

Problems 

12,15,6,9 

,3 

2,14,8,5, 

11 

7,10,13,1 

,4 

5,8,11,14 

,2 

2,5,11,14 

,8 

12,15,6,9 

,3 

3,9,12,15 

,6 

2,8,14,11 

*5 

Figure  23.  Problem  Sequence  by  Documentation  and  Subject 
Number  for  Travis  AFB  Subjects 
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APPENDIX  A 


KEESLER  AFB  PRE-EXPERIMENT 
BIOGRAPHICAL  INFORMATION 


NAME:  

RANK: 

AFSC:  

1.  Date  entered  course  3ABR32834: 

2.  AFSC’ s other  than  32834  held:  

3.  List  any  other  course  work  in  electronics  or  engineering: 

Course  Where  Taken  Credit  Hours  Earned  or 

Weeks  in  Training 


4.  List  any  previous  electronics  or  engineering  job  experience: 
Job  Title  Organization  Length  of  Time 


5.  Have  any  of  your  hobbles  been  electronics  or  engineering  oriented? 

Yes  No  

If  yes,  list  those  hobbles. 


6.  Have  you  ever  used  TOs  for  maintenance  troubleshooting? 

Yes  No  

If  yes,  check  the  number  of  times. 


1-5 
6-10 
over  10 
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7.  Check  the  highest  grade  (or  level)  of  education. 

High  School  nongraduate  

High  School  graduate  

1 year  of  college  

2 years  of  college  

3 years  of  college  

4 years  of  college,  

but  no  degree 

Bachelor ' s Degree  


If  one  or  more  years  of  college,  list  major. 


APPENDIX  B 


FPTA 

MID-EXPERIMENT  QUESTIONNAIRE 


NAME:  

NOTE:  For  questions  1 and  2,  place  an  "X"  along  the  scale  at  the  point 

that  represents  your  opinion  about  the  item  being  discussed.  As 
an  example,  if  the  question  were:  How  do  you  feel  about  vanilla 

ice  cream? 


* 


Don't  like  it.  Like  it  as 

never  eat  it  well  as  any 

other  flavor 


Won ' t eat 
anything 
else 


The  position  of  the  "X"  would  indicate  that  the  individual  answering  the 
question  preferred  vanilla  but  would  eat  other  flavors. 
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How  do  you  feel  about  the  FPTA  as  an  aid  in  troubleshooting  the 
maintenance  problems  you  have  just  had  at  the  following  levels  of 
maintenance? 

COCKPIT  LEVEL 


Not 

useful 
at  all 


Extremely 

useful 


BENCH  LEVEL  TO  SUBASSEMBLIES 


useful 
at  all 


Extremely 

useful 


SUBASSEMBLIES  TO  COMPONENTS 


wot 

Some 

Extremely 

useful 

help 

useful 

at  all 

Comments: 

How  easy  was  it 

to  understand  the  FPTA? 
1 

Very 

1 

No 

Very  easy  to 

difficult 

particular 

understand 

to 

difficulty 

and  follow 

understand 

in  following 

and  follow 

. What  do  you  feel 

are  problems  with  the  FPTA? 

t'W.  >*;>.  *m 


APPENDIX  C 


LTTA 

MID-EXPERIMENT  QUESTIONNAIRE 


NAME: 


NOTE:  For  questions  1 and  2,  place  an  "X"  along  the  scale  at  the  point 

that  represents  your  opinion  about  the  item  being  discussed.  As 
an  example,  if  the  question  were:  How  do  you  feel  about  vanilla 

ice  cream? 

X 

Don't  like  it.  Like  it  as  Won't  eat 

never  eat  it  well  as  any  anything 

other  flavor  else 


The  position  of  the  "X"  would  indicate  that  the  individual  answering  the 
question  preferred  vanilla  but  would  eat  other  flavors. 


1.  How  do  you  feel  about  the  LTTA  as  an  aid  in  troubleshooting  the 

maintenance  problems  you  have  just  had  at  the  following  levels  of 
maintenance? 

COCKPIT  LEVEL 


Not  Some  Extremely 

useful  help  useful 

at  all 

BENCH  LEVEL  TO  SUBASSEMBLIES 


Not  Some  Extremely 

useful  help  useful 

at  all 

SUBASSEMBLIES  TO  COMPONENTS 


Some  Extremely 

help  useful 


Comments: 


Not 

useful 
at  all 


2.  How  easy  was  it  to  understand  the  LTTA? 


Very 

difficult 

to 

understand 
and  follow 


No 

particular 
difficulty 
in  following 


Very  easy  to 
understand 
and  follow 


i 

i 


What  do  you  feel  are  problems  with  the  LTTA? 
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APPENDIX  D 
TO 

MID-EXPERIfJJNT  QUESTIONNAIRE 


NAME:  

NOTE:  For  questions  1 and  2,  place  an  "X"  along  the  scale  at  the  point 

that  represents  your  opinion  about  the  item  being  discussed.  As 
an  example,  if  the  question  were:  How  do  you  feel  about  vanilla 

ice  cream? 


I -1 * 1 

Don't  like  it.  Like  it  as  Won't  eat 

never  eat  it  well  as  any  anything 

other  flavor  else 


The  position  of  the  "X"  would  indicate  that  the  individual  answering  the 
question  preferred  vanilla  but  would  eat  other  flavors. 


? 


I 


i 


i 


!•  do  you  feel  about  the  TOs  as  an  aid  in  troubleshooting  the 

maintenance  problems  you  have  just  had  at  the  following  levels  of 
maintenance? 


COCKPIT  LEVEL 


Not  useful  Some  Extremely 

help  useful 


BENCH  LEVEL  TO  SUBASSEMBLIES 


Extremely 

useful 


Extremely 

useful 


COMMENTS: 


2.  How  easy  was  it  to  understand  the  TOs? 


APPENDIX  E-l 


POST-EXPERIMENT  QUESTIONNAIRE 


(Keesler) 


NOTE:  For  questions  1 through  4,  place  an  "X"  along  the  scale  at  the  point 

that  represents  your  opinion  about  the  item  being  discussed.  As 
an  example,  if  the  question  were:  How  do  you  feel  about  vanilla 

ice  cream? 


Don't  like  it, 
never  eat  it 


Like  it  as 
well  as  any 
other  flavor 


Won't  eat 
anything 
else 


The  position  of  the  "X"  would  Indicate  that  the  individual  answering  the 
question  preferred  vanilla  but  would  eat  other  flavors. 


How  do  you  feel  about  the  FPTA  as  an  aid  in  trouble8hooting  the 
maintenance  problems  you  have  just  had  at  the  following  levels  of 
maintenance? 


COCKPIT  LEVEL 


Hot 

useful 
at  all 


Extremely 

useful 


BENCH  LEVEL  TO  SUBASSEMBLIES 


Not 

useful 
at  all 


Extremely 

useful 


SUBASSEMBLIES  TO  COMPONENTS 


Not 

useful 
at  all 


Extremely 

useful 


How  do  you  feel  about  the  LTTA  as  an  aid  in  troubleshooting  the 
maintenance  problems  you  have  just  had  at  the  following  levels 
of  maintenance? 


COCKPIT  LEVEL 


Not 

useful 
at  all 


useful 


Not 

useful 
at  all 


useful 


SUBASSEMBLIES  TO  COMPONENTS 


Not  Some 
useful  help 
at  all 


Extremely 

useful 


Comments: 


3.  How  easy  was  It  to  understand  the  FPTA? 


Very 

difficult 

to 

understand 
and  follow 


No 

particular 
difficulty 
In  following 


Very  easy  to 
understand 
and  follow 


4.  How  easy  was  it  to  understand  the  LTTA? 


Very 

difficult 

to 

understand 
and  follow 


No 

particular 
difficulty 
In  following 


Very  easy  to 
understand 
and  follow 
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What  do  you  feel  are  problems  with  the  FPTA? 


What  do  you  feel  are  problems  with  the  LTTA? 


Which  troubleshooting  method  was  the  easiest  to  follow? 
Why? 


Would  you  like  to  see  the  Air  Force  use  FPTA  or  LTTA  documentation 
for  any  of  the  following  levels  of  troubleshooting? 


FPTA  - Organizational  Level 

Yes 

No 

- Intermediate  Level 

Yes 

No 

LTTA  - Organizational  Level 

Yes 

No 

- Intermediate  Level 

Yes 

No 

If  you  answered  "MO"  to  any  part  of  question  8 above,  please  tell 
us  why  you  would  not  like  to  see  the  document  used. 

Comments  - 


APPENDIX  E-2 


POST-EXPERIMENT  QUESTIONNAIRE 
(Military  Air  lift  Command) 


NAME:  

NOTE:  For  questions  1 through  4,  place  an  "X"  along  the  scale  at  the 

point  that  represents  your  opinion  about  the  item  being 
discussed.  As  an  example,  if  the  question  were:  How  do  you 

feel  about  vanilla  ice  cream? 

j g 

Don't  like  it.  Like  it  as  Won't  eat 

Never  eat  it  well  as  any  anything 

other  flavor  else 


The  position  of  the  "X"  would  indicate  that  the  individual  answering  the 
question  preferred  vanilla  but  would  eat  other  flavors. 


f 


1. 


How  do  you  feel  about  the  FPTA  as  an  aid  in  troubleshooting  the 
maintenance  problems  you  have  just  had  at  the  following  levels  of 
maintenance? 


COCKPIT  LEVEL 


Not 

useful 
at  all 


Some 

help 


Extremely 

useful 


Not 

useful 
at  all 


Not 

useful 
at  all 


BENCH  LEVEL  TO  SUBASSEMBLIES 


Some 

help 

SUBASSEMBLIES  TO  COMPONENTS 


Extremely 

useful 


Some 

help 


Extremely 

useful 


snts: 


2. 


How  do  you  feel  about  the  LTTA  as  an  aid  in  troubleshooting  the 
maintenance  problems  you  have  just  had  at  the  following  levels 
of  maintenance? 


COCKPIT  LEVEL 


Not 

useful 
at  all 


Some 

help 


Extremely 

useful 


BENCH  LEVEL  TO  SUBASSEMBLIES 


Not 
useful 
at  all 


Some 

help 


Extremely 

useful 
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SUBASSEMBLIES  TO  COMPONENTS 


Not 

useful 
at  all 


Extremely 

useful 


Comments: 


How  do  you  feel  about  the  T.O.  as  an  aid  In  troubleshooting  the 
maintenance  problems  you  have  just  had  at  the  following  levels  of 
maintenance? 

COCKPIT  LEVEL 


Not 

useful 
at  all 


Not 

useful 
at  all 


BENCH  LEVEL  TO  SUBASSEMBLIES 


SUBASSEMBLIES  TO  COMPONENTS 


Extremely 

useful 


Extremely 

useful 


Not 

useful 
at  all 


Extremely 

useful 


4.  How  easy  was  It  to  understand  the  FPTA? 


Very 

difficult 

to 

understand 
and  follow 


No 

particular 
difficulty 
in  following 


Very  easy  to 
understand 
and  follow 


5.  How  easy  was  it  to  understand  the  LTTA? 


Very 

difficult 

to 

understand 
and  follow 


No 

partieu lur 
difficulty 
in  following 


Very  easy  to 
understand 
and  follow 


6.  How  easy  was  it  to  understand  the  T.O.? 


Very 

difficult 

to 

understand 
and  follow 


No 

particular 
difficulty 
in  following 


Very  easy  to 
understand 
and  follow 


7.  What  do  you  feel  are  problems  with  the  FPTA? 


8.  What  do  you  feel  are  problems  with  the  LTTA? 


What  do  you  feel  are  problems  with  the  TO? 


Which  troubleshooting  method  was  the  easiest  to  follow? 
Why? 


Would  you  like  to  see  the  Air  Force  use  FPTA  or  ITTA  documentation 
for  any  of  the  following  levels  of  troubleshooting? 


FPTA  - 

Organizational  Level 

Yes  

No  _ 

- 

Intermediate  Level 

Yes  

No  _ 

LTTA  - 

Organizational  Level 

Yes  

No  _ 

- 

Intermediate  Level 

Yes  

No  _ 

If  you  answered  "NO"  to  any  part  of  question  11  above,  please  tell 
us  why  you  would  not  like  to  see  the  document  used. 

Comments: 
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APPENDIX  F 


INFORMATION  AND  INSTRUCTIONS 


Evaluation  of  Different  Types  of  Technical  Data  for  Troubleshooting 

You  have  been  asked  by  the  Air  Force  to  help  them  in  providing  answers  to 
a number  of  very  important  questions  about  the  suitability  of  two  new  types 
of  technical  data  for  equipment  maintenance  and  troubleshooting. 

The  Air  Force  is  interested  in  providing  manuals  which  assist  the  technician 
in  his  desire  to  do  the  best  possible  job  of  equipment  maintenance.  However, 
without  your  help,  the  Air  Force  will  not  have  the  information  it  needs  to 
make  a correct  decision  about  the  merits  of  each  of  the  proposed  documents. 
For  this  reason,  you  are  being  asked  to  use  each  of  the  documents  in 
troubleshooting  a series  of  carefully  selected  equipment  problems. 

You  are  being  asked  to  work  on  these  problems  in  a closely  controlled 
experimental  evaluation.  In  brief,  the  Air  Force  is  asking  you  to  be  an 
experimental  subject  in  its  field  evaluation  of  the  technical  data.  For  this 
reason,  it  is  important  that  you  follow  all  instructions  given  you. 

One  way  of  comparing  the  merits  of  different  types  of  technical  manuals 
is  to  find  out  how  long  it  takes  to  correctly  isolate  an  equipment  problem 
using  a particular  document. 

Attempts  have  been  made  to  select  problems  which  can  be  located  within  an 
hour  of  troubleshooting.  To  find  out  if  our  estimates  are  correct,  time 
required  to  solve  each  problem  will  be  recorded.  We  would  like  to  emphasize 
that  we  are  NOT  interested  in  finding  out  how  quickly  YOU  can  solve  the 
problem;  we  want  to  know  the  length  of  time  it  takes  the  proposed  technical 
document  to  guide  you  to  the  correct  solution.  YOU  ARE  NOT  BEING  TESTED  — 
THE  DOCUMENT  IS! 


APPENDIX  G 


EXAMPLE 

SPECIFIC  INSTRUCTIONS 

1.  Approximately  one  hour  is  allocated  to  each  troubleshooting  problem. 
There  is  more  than  enough  time  for  each  problem,  so  do  not  rush. 

2.  The  technical  document  you  are  using  is  intended  to  be  self-explanatory. 
If  you  have  a problem,  read  the  item  in  question  again.  If  after 
repeating  the  item  you  still  have  a problem,  then  ask  the  experimenter. 

3.  Notify  the  experimenter  of  any  decisive  steps  (removal  of  parts  or 
branching  from  steps)  BEFORE  making  them. 

4.  Tools  not  furnished  will  be  supplied  upon  request  if  required. 

5.  When  time  is  started,  you  are  to  turn  this  page,  read  the  work  order 
given,  and  begin  the  troubleshooting  procedures. 

6.  Manuals  for  the  test  equipment  are  available  for  you  to  use  if 
necessary. 

7.  During  this  exercise,  observe  all  of  the  standard  safety  procedures. 

8.  The  experimenter  will  inform  you  when  the  time  is  up. 


DO  NOT  TURN  THIS  PAGE  UNTIL 
INSTRUCTED  TO  DO  SO  BY  THE 
EXPERIMENTER. 


WORK  ORDER 


Troubleshoot  malfunctioning  Radar  Receiver-Transmitter  down  to  faulty 


subassembly 
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